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Abstract —The least-mean-squares (LMS) algorithm is the most 
popular algorithm in adaptive filtering. Several variable step-size 
strategies have been suggested to improve the performance of 
the LMS algorithm. These strategies enhance the performance 
of the algorithm but a major drawback is the complexity in 
the theoretical analysis of the resultant algorithms. Researchers 
use several assumptions to find closed-form analytical solutions. 
This work presents a unified approach for the analysis of variable 
step-size LMS algorithms. The approach is then applied to several 
variable step-size strategies and theoretical and simulation results 
are compared. 

Index Terms - Variable step-size, least-mean-square algo¬ 
rithms 

I. Introduction 

Many algorithms have been proposed for estimation/system 
identification but the LMS algorithm has been the most 
popular in as it is simple and effective. However, a limit¬ 
ing factor of LMS is that if the step-size of the algorithm 
is kept high then the algorithm converges quickly but the 
resultant error floor is high. On the other hand, lowering 
the step-size is results in improvement in the error perfor¬ 
mance but the speed of the algorithm becomes slow. In order 
to overcome this problem, various variable step-size (VSS) 
strategies have been suggested, which have a high step-size 
initially for fast convergence but then reduce the step-size 
with time in order to achieve a low error performance Ei- 
ll25l . Some algorithms are proposed in literature for specific 
applications llT2ll . irT4l . llT5ll . ll20l - ll25l . There are several algo¬ 
rithms that are derived from a constraint on the cost function 

111,121,Hi,Ho),m. 

In general, all VSS algorithms aim to improve performance 
at the cost of computational complexity. This trade-off is 
generally acceptable due to the improvement in performance. 
However, the additional complexity also results in difficulty 
in analyzing the algorithm. Authors use several basic assump¬ 
tions to find closed-form solutions for the analysis of these 
algorithms. Most of these assumptions are similar. However, 
each algorithm has to be dealt with separately in order to 
find the steady-state misadjustment, leading to the steady-state 
excess-mean-square-error (EMSE). Similarly, the mean-square 
analysis for each algorithm has to be performed individually. 

Based on the similarity of the assumption used by the 
authors of all these VSS algorithms, this work presents a 
unified approach for the analysis of VSS LMS algorithms. 
The aim of this work is to perform a generalized analysis 
for any VSS strategy that is based on the LMS algorithm. 
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This analysis can be applied to most existing as well as any 
forthcoming VSS algorithms. 

The rest of the paper is divided as follows. Section II 
presents a working system model and problem statement. 
Section III details the complete theoretical analysis for VSS 
LMS algorithms. Simulation results are presented in section 
IV. Section V concludes this work. 

H. System Model 

The unknown system is modeled as an EIR filter in the form 
of a vector, Wo, of size (M x 1). The input to the unknown 
system at any given time i is a (1 x M) complex-valued 
regressor vector, u(i). The observed output of the system is a 
noise corrupted scalar, d{i). The variables of the system are 
related by 

d{i) = u{i)sVo + v{i), (1) 

where v{i) is the complex-valued zero-mean additive noise. 

The LMS algorithm iteratively estimates the unknown sys¬ 
tem with an update equation given by 

w(i-f 1) = w(i)-b/re(i)u*(i), (2) 

where w(i) is the estimate of the unknown system vector at 
time i, e{i) = d{i) — u(i)w(j) is the instantaneous error and 
(.)* is the complex conjugate transpose operator. The step-size 
for the update is defined by the variable /r, which is fixed for 
the LMS algorithm. In case of a variable step-size algorithm, 
the step-size is also updated iteratively. The VSS LMS update 
equations are given by 

w(i-b 1) = w(i)-b/r(z)e(z)u*(z), (3) 

m(* + 1) = /{m(*)}, (4) 

where /{.} is a function that defines the update equation for 
the step-size and is different for every VSS algorithm. 

While performing the analysis of the LMS algorithm, the 
input regressor vector is assumed to be independent of the 
estimated vector. Eor the VSS algorithms, it is generally 
assumed that control parameters are chosen such that the 
step-size and the input regressor vector are asymptotically 
independent of each other, resulting in a closed-form steady- 
state solution that closely matches with the simulation results. 
Eor some VSS algorithms, the analytical and simulation results 
are closely matched during the transition stage as well but this 
is not always the case. The results are still acceptable for all 
algorithms as a closed-form solution is obtained. 

The main objective of this work is to provide a generalized 
analysis for VSS algorithms, in lieu with the assumptions 
mentioned above. The results of this analysis can be applied to 
VSS algorithms in general as will be shown through specific 
examples. 
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III. Proposed analysis 
The weight-error vector is given by 

w(j) = Wo - w( 2 ). (5) 

Using 0 in (O results in 

w(i + l) = [Im - M(i)u*(i)u(j)]w(2) 

( 6 ) 

where Im is an identity matrix of size M. Before beginning 
the analysis, another assumption is made, without loss of 
generality. The input data is assumed to be circular Gaussian. 
As a result, the auto correlation matrix of the input regressor 
vector, given by Rti = E[u*(i)u(i)], where E[.] is the 
expectation operator, can be decomposed into its component 
matrices of eigenvalues and eigenvectors, R„ = TAT*, 
where T is the matrix of eigenvectors such that T*T = Im 
and A is a diagonal matrix containing the eigenvalues. Using 
the matrix T, the following transformations are made 

w(i) = T*w(i) u(i) = u(*)T 

The weight-error update equation thus becomes 

w(i + 1) = [Im - At( 2 )u*(*)u( 2 )]w(z) - p.(i)u*(i)v(i). (7) 

A. Mean Analysis 

Applying the expectation operator to 0 results in 
E [w(i -f 1)] = E [{Im - 

= {Im -E[/^(i)u*(i)u(i)]}E[w( 2 )], (8) 

where the data independence assumption is used to separate 
E[w(j)] from the rest of the variables. The second term is 
0 as additive noise is independent and zero-mean. Using the 
assumption that the step-size control parameters are chosen in 
such a way that the step-size and the input regressor data are 
asymptotically independent, ® is further simplified as 

E[w(i-fl)] = {Im - E [^(j)] A}E [w(j)], (9) 

where E [u*(*)u(i)] = A. The sufficient condition for stability 
is evaluated from 0 and is given by 

0<E[/x(2)] < (10) 

Pmax 

where Pmax is the maximum eigenvalue of A. 


B. Mean-Square Analysis 

Taking the expectation of the squared weighted Z 2 -norm of 
O yields 


E 


|w(2 -f 


( 11 ) 


= E [w*(2)S'w(2)] -f E [/2^(z)u^(2)u(z)Su*(i)] 
-E [^(i)u(i)u(i)S {Im - ^(i)] 

-E [w*(i) {Im - At(*)u*(i)u(0} , 


where ||.|| is the ( 2 -norm operator and S is a weighting matrix. 
The weighting matrix S' is given by 


S' = {Im - S {Im - 

= Im — / 2 (j)u*(i)u(i)S — /r( 2 )Su* ( 2 ) 0 ( 2 ) 

-|-/ 2 ^( 2 )u*( 2 )u( 2 )Su*( 2 )u( 2 ) (12) 


The last term in (fTTT i is 0 due to independence of additive 
noise. Using the data independence assumption, the remaining 
2 terms are simplified as 


E[||w (2 + l)|[y =e[||w( 2 )|[^,J +a2E[p2(2)]Tr{SA}, 

(13) 

where is the additive noise variance, Tr{.} is the trace op¬ 
erator and E [u( 2 )Su^( 2 )] = Tr{SA}. Once again invoking 


the data independence assumption, we write E 


]^|w(i)llE[S' 

(Ha is rewritten as 


|w(() 


. Further, taking E [S'] = S' and simplifying. 


= Im — 2E [n{i)] AS -f E ATr [SA] 

+E [n^{i)] ASA. (14) 

Using the diag operator, (fTsT l and (fT4li are simplified as 


E 


l|w(2-(-i)|[; 


= E 


|w(i) 


lF(i)<T 


+ alE[n^{i)] (15) 


where cr = diag {S}, A = diag{A}, the weighting matrix S' 
is replaced with diag {S'} = cr' = F( 2 )cr, where F( 2 ) is 


Fil)=lM-2E[^i{l)]A + E[^i\i)] 


\y 


(16) 


Now, using (flSl l and (fTdl l. the analysis iterates as 


E llw 


2 II l|2 

= Wo L 


F(0) =Im-2a2(0)A + a2"(0) 


A^ + AA^ 


where E [/u(0)] = fi{0) and E [/i^(0)] = /2^(0) as this is the 
initial step-size value. The first iterative update is given by 


E 


|w(l)|[^ 


= E 


|w(0)|| 


2 

F(0)c 


+ <t„V(o)a^ 


'^o|If(0)ct 


+ a,V(0)AM 


F(l) = Im-2E H1)]A + E[a22(1)] 


A^ -f AA" 


where the updates E [/^(l)] and E [/i^(l)] are obtained from 
the particular step-size update equation of the VSS algorithm 
being used. Similarly, the second iterative update is given by 


E 


|w(2)||; 


= E 


= w, 


|w(l)|| 


F(1)<t 


+ a^E [^^(1)] A’ 


oIIf(i)f(o)<t 3”(6)A F(l)t 


-fcr^E [/2^(1)] A^ 


= 11^0 

+a2A^{p2(0)F(l)+E[p2(l)] I^}, 


''o|If(1)F(0)ct 


F(2) = Im - 2E [^(2)] A + E [fi^2)] 


A^ + AA" 
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Continuing, the third iterative update is given by 


E 


|w(3)||; 


= E 


= llw, 


w(2)||f(2)o-J [^2(2)] A cr 

F(2)A(2)<t + [m (2)] cr 
ft fc+1 'j 

+al\^ HfM U 




F(3) = Im - 2E [/i(3)] A + E [^"(3)] 


AA" 


where the weighting matrix A(2) = F(1)F(0). The fourth 
iterative update is then given by 


E 


(4)11. =l|wo||^(3)A(3)^ + a2E[M2(3)]A^ 


fc+1 


s XI ® (^)] n ^ ("^) f 




m—3 
2 / 


F(4) = Im - 2E [/i(4)] A + E [^"(4)] 


AA" 


where the weighting matrix A(3) = F(2)A(2). Now, from the 
third and fourth iterative updates, we generalize the recursion 
for the update as 


E 


r{i) 


= l|Wo|lF(i 


i-l)A(i-l)<T + “ 4)] 


fe+1 


Ta^A^ X E [m" (fc)] n F (m) <T (17) 




m—i—1 


F(z) =Im-2E [^i{^)]A + E[^i^{i)] 


\V 


(18) 


Similarly, the recursion for the (i + 1)*^ update is given by 


E 


w(z +: 


= w„ 


.)aw. + ^'E [^^\^)] A^<t(19) 

( 2—1 fe+l ^ 

) X ® (^)] n ^ r 




F(2 + 1) = Im- 2E [^(^ + l)]A 


+E [^^(z + 1)] 


AA' 


( 20 ) 


Subtracting (fTTl i from ( fTOl l and simplifying the terms gives the 
final recursive update equation 


E 


|w(z - 


= E 


|w(i) 


+ IIwoIIfwaw. + 


+a2A^{F(z)-lM}B(z)<T, 


( 21 ) 


where 


fc+i 


B{^) = \E[^i^{l-l)]lM + J2E[^i^k)] n F(m) . 


k^O 


m—i— 1 


( 22 ) 

The final set of iterative equations for the mean-square 
learning curve are given by (l2Tl l. (fTSl l and 

A(z + 1) =F(z)A(i) (23) 

B(z + 1) = E [/z"(z)] Im + F(i)B(z). (24) 


Taking the weighting matrix S = Im results in the mean- 
square-deviation (MSD) while taking the weighting matrix 
S = A gives the EMSE. 

It should be noted here that unlike the analysis given in 
HI for the LMS algorithm, the weighting matrix F(z) is 
not constant. As a result, the Cayley-Hamilton theorem is 
not applicable. In this context, (fTSl) and (l2TI) - (l24l) are very 
significant contributions of this work. 


C. Steady-State Analysis 

At steady-state, the recursions ( fTSl) and (fThb become 


E 


= E 

Fss = Im 2^ssA 


ssWl^ 


I 2 2 \T 

+ (T 


\y 


(25) 

(26) 


where the subscript ss denotes steady-state. Simplifying ( l25b 
further gives 


E 


2 2 \1 


Im — Fg 


1-1 


(27) 


which defines the steady-state MSD if S = Im and steady- 
state EMSE if S = A. 


D. Steady-State Step-Size Analysis 

The analysis presented in the above section has been generic 
for any VSS algorithm. In this section, 5 different VSS algo¬ 
rithms are chosen to present the steady-state analysis for the 
step-size. These steady-state step-size values are then directly 
inserted into (l27l) and (l26l l. The 5 different VSS algorithms 
and their step-size update equations are given in Table U The 
first algorithm, denoted KJ is the work of Kwong and Johnston 
0. The second algorithm, denoted by AM, also refers to the 
authors Aboulnasr and Mayyas 0. The NC algorithm refers to 
the noise-constrained LMS algorithm ifTOl . The VSQ algorithm 
is the variable step-size quotient LMS algorithm, based on the 
quotient form ifTSll . The last algorithm, denoted by Sp, refers 
to the Sparse VSSLMS algorithm of Il22l . 


Algorithm 

Step-size update equation 

ftlE] 

(i -1- 1) = Oki/i (i) -y ■ykie^ (i) 

AM y 

P (i) = PamP (i — 1) -1- (1 — dam) e (i) e (i — 1) 

/I (i -1- 1) = aarnkt (i) -1- lamP^ (*) 

NC llOl 

f-i {i 1) = /.iO (1 'ync07ic{i + 1)) 

0nc{i + 1) = (1 - anc)0nc{i) + (e^(^) - ^v) 

VSQ (H 

Aq{i) = aqAq{i — 1) -1- e^(j) 

Bqli) = bqBq{i — 1) + 6^ (z) 
a z,-\ — ^g(z) 

Zj(z-|- 1) = aqfl(i)-\--fqeq (z) 

Sp (22l 

/z (z -1- 1) = OspP (z) -1- 'Yap |e (z)| 


TABLE I 

Step-size update equations for the VSSLMS algorithms. 


Applying the expectation operator to the step-size update 
equations and simplifying gives the equations presented in 
Table ID 

At steady-state, the expected step-size E [p, (z)] is replaced 
by Pss- The approximate steady-state step-size equations are 
given in Table HD The steady-state EMSE (denoted by in 
the tables) value is assumed to be small enough to be ignored. 
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Algorithm 

Expectation of update equation 

fimj 

E [/I (i -1- 1)] = afe,'E [/i (i)] -|- 'yki |C(*) + ■ 

AM (6) 

E [p^ (i)J = [p (j - 1)] + (1 - PamY (C(*) + 

■ (C(* - 1) + 

E[lJ.(i+ 1)] = OamE [p, (i)] -1- 7amE 

NC 1101 

E I6nc{i + 1)1 = (1 - anc)E [0nc(i)\ + o^nct{i)/2 

E [/i (i H- 1)] = /TO (1 H" TncFj [Onci'i H- 1))] 

VSQ (H 

TTr/.U-j-lM n IT r,, UAl a 1) 

+ai)+Y 



sp da 

E [p (i -1- 1)] = OspE [p (f)] -1- 'yapY^ull-K 


TABLE II 

Expected values eor the update equations from Table[T] 


Algorithm 

Steady-state step-size value 

KJ 0 

P-ss ^ — (j^ 

AM (6) 

~ 2 

pss ~ l-OLn.-m. 

NC llOl 

Pss ~ PO • 

VSQ [H 

Mss ~ l-c«„(l-a<,) ■ 

Sp da 

Mss ~ A- 


TABLE III 

Steady-state step-size values for equations from Table[I] 


IV. Results and Discussion 

In this section, the analysis presented above will be tested 
upon the 5 VSS algorithms listed in Table H] These algorithms 
are used in 2 different experiments to test the validity of the 
analysis. In the first experiment, MSD is plotted using (ISTT i 
and compared with simulation results. The second experiment 
compares the steady-state simulation results with the theoret¬ 
ical results obtained using dZTl i. 

For the first experiment, the length of the unknown vector 
is M = 4. The input regressor vector is a realization of a 
zero-mean Gaussian random variable with unit variance. The 
signal-to-noise ratio (SNR) is chosen to be 20 dB. The step- 
size control parameters chosen for this experiment are given 
in Table |IV] The results are shown in Fig. [T] As can be seen 
from the figure, there is a close match between simulation and 
analytical results for all the algorithms. 


Algorithm 

Parameters 

Ejia 

ciki = 0.995, 7 fc,' = le — 3 

AM y 

Pam = 0.9, Oiam = 0.995, 7am = le - 3 

NC llOl 

'^nc — 10, Cx.nc — Ic — 3 

VSQ list 

a = 0.99, bq = le — 3, aq = 0.995, "fq = le — 3 

Sp d2) 

Clap = 0.995, 7 sp = le — 3 


TABLE IV 

Step-size CONTROL parameters for the VSSLMS algorithms. 


The second experiment compares simulation results for 
the VSS algorithms with theoretical steady-state MSD results 
obtained using ( |27] ). The step-size control parameters are 
chosen the same as the previous experiment. The results for 
this experiment are given in Table |V] It can be seen that there 
is an excellent match between theory and simulation results. 



Fig. 1. Theory (21) V simulation MSD comparison for 5 different VSS 
algorithms . 


Algorithm 

MSD (dB) 
equation 1271 

MSD (dB) 
simulation 

Km 

-43.96 

-43.94 

AM 0 

-46.98 

-46.98 

NC 1101 

-29.42 

-29.26 

VSQ 1181 

-33.76 

-33.77 

Sp 1221 

-34.78 

-34.72 


TABLE V 

Theory v simulation comparison for steady-state MSD for 
DIFFERENT VSSLMS ALGORITHMS. 


V. Conclusion 

This work presents a unified approach for the theoretical 
analysis of LMS-based VSS algorithms. The iterative recur¬ 
sions presented here differentiate this work from previous 
analyses in that this set of equations provides a generic 
treatment of the analysis for this class of algorithms for the first 
time. Simulation results confirm the generic behavior of the 
presented work, for both the transient state as well as steady- 
state. 
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